Lack of labeled data accumulated for telecommunication fraud in the bank and high cost of manually labeling cause the insufficiency of labeled data that can be used in supervised learning methods for telecommunication fraud detection. To solve this problem, an unsupervised learning method based on dense subgraph was proposed to detect telecommunication fraud. Firstly, subgraphs with high anomaly degree in the network of accounts and resources (IP addresses and MAC addresses) were searched to identify fraud accounts. Then, a subgraph anomaly degree metric satisfying the features of telecommunication fraud was designed. Finally, a suspicious subgraph searching algorithm with resident disk, efficient memory and theory guarantee was proposed. On two synthetic datasets, the F1-scores of the proposed method are 0.921 and 0.861, which are higher than those of CrossSpot, fBox and EvilCohort algorithms while very close to those of M-Zoom algorithm (0.899 and 0.898), but the average running time and memory consumption peak of the proposed method are less than those of M-Zoom algorithm. On real-world dataset, F1-score of the proposed method is 0.550, which is higher than that of fBox and EvilCohort while very close to that of M-Zoom algorithm (0.529). Theoretical analysis and simulation results show that the proposed method can be applied to telecommunication fraud detection in the bank effectively, and is suitable for big datasets in practice.